Safety system for a vehicle to detect and warn of a potential collision

ABSTRACT

A system mountable in a vehicle to provide object detection in the vicinity of the vehicle. The system includes a camera operatively attached to a processor. The camera is mounted externally at the rear of the vehicle. The field of view of the camera is substantially in the forward direction of travel of the vehicle along the side of the vehicle. Multiple image frames are captured from the camera. Yaw of the vehicle may be input or the yaw may be computed from the image frames. Respective portions of the image frames are selected responsive to the yaw of the vehicle. The image frames are processed to detect thereby an object in the selected portions of the image frames.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalApplication No. 62/162,838, filed May 18, 2015, and U.S. ProvisionalApplication No. 62/261,759, filed Dec. 1, 2015. This application alsoclaims the benefit of prior U.K. Application No. GB 1511316.0, filedJun. 29, 2015. All of the foregoing applications are incorporated hereinby reference in their entirety.

BACKGROUND

1. Technical Field

The present disclosure relates to a safety system for a vehicle todetect and warn of a potential collision with an object in the path oftravel of the vehicle as part of a driver assistance system.

2. Background Information

Driver assistance systems (DAS) typically include lane departure warning(LDW), Automatic High-beam Control (AHC), pedestrian recognition,forward collision warning (FCW) and pedestrian detection. Vehicledetection, pedestrian detection and traffic sign recognition algorithmsmay share a common general structure: an initial candidate detectionfollowed by more computationally intensive classification of the initialcandidate. Obstacle detection may also be performed using variousstructure-from-motion algorithms.

Lane departure warning (LDW) systems are designed to give a warning inthe case of unintentional lane departure. The warning is given when thevehicle crosses or is about to cross the lane marker. Driver intentionis determined based on use of turn signals, change in steering wheelangle, vehicle speed and brake activation.

Traffic sign recognition (TSR) modules are designed typically to detectspeed limit signs and end-of-speed limit signs on highways, countryroads and urban settings. Partially occluded, slightly twisted androtated traffic signs are preferably detected. Systems implementingtraffic sign recognition (TSR) may or should ignore the following signs:signs on truck/buses, exit road numbers, minimum speed signs, andembedded signs. A traffic sign recognition (TSR) module which focuses onspeed limit signs may not have a specific detection range requirementbecause speed limit signs only need to be detected before they leave theimage.

The core technology behind forward collision warning (FCW) systems andheadway distance monitoring is vehicle detection. A key component of atypical forward collision warning (FCW) algorithm is the estimation ofdistance from a single camera and the estimation of scale change fromthe time-to-contact/collision (TTC) as disclosed for example in U.S.Pat. No. 7,113,867.

Structure-from-Motion (SfM) refers to methods for recoveringthree-dimensional information of a scene that has been projected ontothe back focal plane of a camera. The structural information derivedfrom a SfM algorithm may take the form of a set of projection matrices,one projection matrix per image frame, representing the relationshipbetween a specific two-dimensional point in the image plane and itscorresponding three-dimensional point. SfM algorithms rely on trackingspecific image features from image frame to image frame to determinestructural information concerning the scene.

SUMMARY

Various systems and methods are disclosed herein performable by a systemmountable in a vehicle to provide object detection in the vicinity ofthe vehicle. The system includes a camera operatively attached to aprocessor. The camera is mounted externally at the rear of the vehicle.The field of view of the camera is substantially in the forwarddirection of travel of the vehicle along the side of the vehicle.Multiple image frames are captured from the camera. Yaw of the vehiclemay be input or the yaw may be computed from the image frames.Respective portions of the image frames are selected responsive to theyaw of the vehicle. The image frames are processed to detect thereby anobject in the selected portions of the image frames. The yaw ismeasurable or determinable by processing the image frames, or input froma gyroscopic device, turn signals, steering angle and a sensor attachedto the steering column of vehicle. Upon measuring an increase inabsolute value of the yaw, a larger area of the image frames isprocessed to increase an effective horizontal field of view and todetect thereby an obstruction over a wider horizontal angle measuredfrom the side of the vehicle.

For a camera installed on the right side of the vehicle, when thevehicle turns right, a larger area of the image frames is processed toincrease an effective horizontal field of view and to detect anobstruction over a wider horizontal angle measured from the right sideof the vehicle.

For a camera installed on the left side of the vehicle, when the vehicleturns left, a larger area of the image frames is processed to increasean effective horizontal field of view and to detect an obstruction overa wider horizontal angle measured from the left side of the vehicle. Itmay be determined that the object is a moving obstacle and verified thatthe object will be in a projected path of the vehicle to determine apotential collision of the vehicle with the moving obstacle. The driveris warned of the potential collision of the vehicle with the movingobstacle. The warning may be sounded with an audible warning and/ordisplayed with a visible warning. The warning may issue from the side ofthe vehicle on which is mounted the camera viewing the moving obstacle.The object may be a bicyclist, a motorcyclist, a vehicle, a pedestrian,a child riding on a toy vehicle, a person with pram and a wheelchairuser. Another driver assistance system: lane detection, structuralbarrier recognition and/or traffic sign recognition may be used torecognize a stationary object, and the warning to the driver may besuppressed if all objects detected in the immediate vicinity of thevehicle are stationary objects. The stationary objects may include astructural barrier, lane marking, a traffic sign, tree, wall and/or apole.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed embodiments are herein described, by way of example only,with reference to the accompanying drawings, wherein:

FIGS. 1A and 1B show a side view and a plan view respectively of avehicle, according to an aspect of the disclosed embodiments.

FIG. 2A shows a block diagram of a system, according to an aspect of thedisclosed embodiments.

FIG. 2B shows a view from inside a vehicle, according to a feature ofthe disclosed embodiments.

FIG. 3 shows a flowchart of a method, according to an aspect of thedisclosed embodiments.

FIGS. 4A and 4B, show diagrams of road scene scenarios, according to anaspect of the disclosed embodiments.

FIG. 5A shows a driver's forward view from inside a vehicle, accordingto features of the disclosed embodiments.

FIGS. 5B, 5C, 5D and 5E show examples of a center warning display,according to features of the disclosed embodiments.

FIG. 5F shows right and/or left warning displays, according to a featureof the disclosed embodiments.

FIG. 6 shows a flowchart for a method, according to aspects of thedisclosed embodiments.

The foregoing and/or other aspects will become apparent from thefollowing detailed description when considered in conjunction with theaccompanying drawing figures.

DETAILED DESCRIPTION

Reference will now be made in detail to exemplary embodiments, examplesof which are illustrated in the accompanying drawings, wherein likereference numerals refer to the like elements throughout. Theembodiments are described below by referring to the figures.

By way of introduction, it is well known that a driver of a bus or of aheavy vehicle has limited visibility in the immediate vicinity of thevehicle. Particularly, the pillars which separate the front window fromthe sides of the bus partially blocks the view of the driver. Moreover,the driver has limited field of view at the sides of the vehicle whichmay be in part compensated for by placement of side mirrors. As such,the driver may be unaware of potential obstacles, e.g., bicyclists,pedestrians, which move into “blind spots” not readily visible by thedriver. Disclosed embodiments may provide a warning to the driver when amoving obstacle encroaches to the side of the vehicle especially whileturning to avoid a collision between the unseen moving obstacle and theside of the vehicle. According to a feature of the disclosedembodiments, the area of interest in the image frames and the effectivehorizontal field of view or angle of interest measured from the side ofthe bus changes when the vehicle is turning. When the vehicle is turningright, the angle of interest on the right side increases and when thebus is traveling straight after the turn the horizontal angle ofinterest decreases again. The system is configured to process the imageframes and detect potential obstacles in the area of interest of theimage frames.

Decreasing the area of interest when the vehicle is traveling straightis important to reduce false positive warnings to the driver when thebus is traveling straight and there is less likelihood of collision thanduring a turn.

A driver of a bus pulling out of a bus stop is normally more attentiveto the lane of traffic with which he/she is trying to merge, in the USAfor instance to the left of the bus and in

Great Britain to the right of the bus, rather than any obstacles whichmay be adjacent with the sidewalk. If a pedestrian, bicyclist or a childon a toy vehicle falls unexpectedly from the sidewalk into the path ofthe bus, then an injury or fatality may result. Similarly, the driver ofthe bus while viewing oncoming traffic through the rear view mirror maybe unaware of a pedestrian or bicyclist which moved in front of the bus.Thus, according to another feature of the disclosed embodiments, anaudible and/or visible warning to the driver calls his attention to arecognized obstacle near the path of the bus of which he/she wouldotherwise be unaware.

In some embodiments, the disclosed systems and methods may beimplemented to navigate an autonomous vehicle. As used throughout thisdisclosure, the term “autonomous vehicle” refers to a vehicle capable ofimplementing at least one navigational change without driver input. A“navigational change” refers to a change in one or more of steering,braking, or acceleration of the vehicle. To be autonomous, a vehicleneed not be fully automatic (e.g., fully operation without a driver orwithout driver input). Rather, an autonomous vehicle includes those thatcan operate under driver control during certain time periods and withoutdriver control during other time periods. Autonomous vehicles may alsoinclude vehicles that control only some aspects of vehicle navigation,such as steering (e.g., to maintain a vehicle course between vehiclelane constraints), but may leave other aspects to the driver (e.g.,braking). In some cases, autonomous vehicles may handle some or allaspects of braking, speed control, and/or steering of the vehicle.

Reference is now made to FIGS. 1A and 1B which show a side view and aplan view respectively of a vehicle 18, according to an aspect of thedisclosed embodiments. Vehicle 18 may be a public transport bus, schoolbus, coach, a heavy goods vehicle (HGV), or a passenger vehicle (e.g., acar, such as a sedan). In some embodiments, vehicle 18 may be longerthan a typical vehicle (e.g., such as a limousine, truck, etc.). Mountednear the rear of vehicle 18 on the right and left sides respectively ofvehicle 18 are two cameras 12R and 12L. Cameras 12R and 12L are mountedso that the respective fields of view of cameras 12R and 12L includegenerally the forward direction of travel of vehicle 18 shown by dottedlines. A camera 12C is shown mounted at the front of vehicle 18 with afield of view which is generally centered in the forward direction oftravel of vehicle 18 shown by dotted lines. Cartesian axes X,Y,Z areshown. Axis Z is the long horizontal axis along the length of vehicle18. Axis Y is the vertical axis of vehicle 18. Rotation around axis Ydefines yaw of vehicle 18 and/or of cameras 12. Yaw is controlled by thesteering mechanism of vehicle 18. Axis X is horizontal along the widthof vehicle 18.

Reference is now also made to FIG. 2A which shows a block diagram of asystem 16, which includes camera 12 one of cameras 12C, 12R or 12Lshowing various features of the disclosed embodiments. Differentinstances of system 16 may be attached to left camera 12L right camera12R and center camera 12C which are not separately illustrated for thesake of brevity. Processor 14 receives and processes image frames 15from camera 12. A storage or memory 20 connects bidirectionally toprocessor 14. Memory 20 may be used to store the algorithms of camerabased driver assistance systems (DAS) known in the art which providepedestrian and/or other class-based object, e.g. bicycle, motorcycle,detection/recognition and/or general obstacle detection/recognition. Anoutput from processor 14 is shown attached to a display 32. An input toprocessor 14 from a yaw sensor is shown. System 16 may be installed inmultiple places in vehicle 18 as shown in FIGS. 1A and 1B.Alternatively, multiple cameras 12 for instance side cameras 12R and 12Land center camera 12C may be connected to and provide multiple sets ofimage frames respectively to a single processor 14.

Reference is now made to FIG. 2B which shows a view 30 from insidevehicle 18, according to a feature of the disclosed embodiments. View 30shows the back of a driver 36 as driver 36 operates vehicle 18. To theleft of driver 36 is a display 32L which is used to provide a visibleand/or audible warning to driver 36 of an object detected and/orrecognized visible to left camera 12L and recognized if the object is inpossible collision with vehicle 18. The detection/recognition of theobject is provided by one of processors 14L/14R processing image frames15L/15R captured by cameras 12L/12R.

In some embodiments, as an alternative to providing a warning to driver36, system 16 may instead cause the vehicle (e.g., vehicle 18) to takean action. For example, system 16 may analyze the collected data todetermine whether or not the vehicle (e.g., vehicle 18) should take acertain action, and then automatically take the determined actionwithout human intervention. For example, in response to recognizing thatan object is in a possible collection with the vehicle, system 16 mayautomatically control the braking, acceleration, and/or steering of thevehicle (e.g., by sending control signals to one or more of a throttlingsystem, a braking system, and a steering system). In yet otherembodiments, system 16 may provide a visible and/or audible warning tothe driver and cause the vehicle to take an action.

Similarly, to the right of driver 36 is a display 32R which is used toprovide a possible warning to driver 36 of a potential collision with anobject to the right of driver 36 and/or cause the vehicle to take anaction, such as automatically controlling the braking, acceleration,and/or steering of the vehicle (e.g., by sending control signals to oneor more of a throttling system, a braking system, and a steeringsystem). In front view of driver 36 is a display 32C which providesvisible and/or audible notification of possible detection of an objectto the front of driver 36 in image frames 15C captured by camera 12C.Displays 32L, 32C and 32R are positioned to provide a display to driver36 which follows the view of driver 36 as he/she drives vehicle 18. Forexample, as driver 36 drives straight ahead, display 33C is positionedin his/her general view, whereas if turning left or right displays 33Land 33R respectively are positioned in his/her general view.

Reference is now also made to FIG. 3 which shows a flowchart of a method301, according to an aspect of the disclosed embodiments using system 16which includes multiple cameras 12C, 12R and 12L mounted on vehicle 18as in FIGS. 1A and 1B. Cameras 12R, 12L and 12C independently provideimage frames 15R, 15L and 15C respectively which are captured (step 303)by respectively by processors 14C, 14L and 14R. Yaw information 4 may beinput (step 305) into processors 14C, 14L and 14R. Yaw information 4 maybe derived from any or combination thereof of a yaw sensor, e.g.gyroscopic device, use of turn signals or change in steering wheel anglevia the CAN bus of vehicle 18 or a sensor attached to the steeringcolumn of vehicle 18 which provides a measure of steering column angularposition/velocity as vehicle 18 turns left or right. Yaw information 4may be derived alternatively or in addition from processing image motionin image frames 15. For example, in some embodiments, yaw information 4may be derived by analyzing at least two image frames. The analysis mayinclude identifying an object in the at least two image frames anddetermining that the object has shifted or moved position. Thisdetermination may compare, for example, the position of the objectrelative to a center point or edge of the at least two image frames.

In step 307, the processor sets an area of interest in image frame 15responsive to yaw information 4. For instance, if vehicle 18 is turningright, then processor 14R is configured to use a wider portion(nominally in horizontal image coordinates) of images frames 15R forsubsequent detection and classification (step 309). If vehicle 18 isturning left, then processor 14L is configured to use a wider portion ofimages frames 15L for subsequent detection and classification (step309). If vehicle 18 is traveling straight then processors 14R and 14Lare configured to use a narrower portion (in horizontal imagecoordinates). In decision block 309 algorithms stored in memory 20R/20Land/or included in circuitry of processors 14C/14R/14L are used todetect and/or recognize obstacles such as pedestrians, motorcycles,bicycles and general obstacles, the images of which are at least in partin the area of interest responsive to the yaw information. If such anobject is detected and/or recognized then driver 36 is warned (step313). Otherwise, method 301 continues with capture of multiple imageframes (step 303). It is advantageous to set (step 307) the area ofinterest in image frames 15R/15L responsive to yaw information (step307) to reduce false positive object detection/classification in thevicinity of vehicle 18 and to avoid frequently warning (step 313) driver36 when there is no expectation for an object detection/classificationin the area of interest (decision box 309).

In some embodiments, in addition to or as an alternative to using yawinformation, system 16 may determine to use a portion of one or moreimage frames (e.g., a wider or narrower portion) based on at leastinformation stored in a local or remote database. For example, the storeinformation may include information collected on prior navigations of aparticular area or location. This information may include areas withlikely obstacles, such as pedestrians, and the information may specificthe portion of the image frames to use in such areas. In someembodiments, the portion of an image frame to use for subsequentdetection and classification may be chosen based on stored map dataand/or based on a location of the vehicle determined via GPS data. Forexample, based on a location of the vehicle, system 16 may determine touse a certain portion of one or more image frames for subsequentdetection and classification.

Reference is now made to FIGS. 4A and 4B, which shows a diagram of aroad scene scenario 40 a and 40 b respectively, according to aspects ofthe disclosed embodiments. In FIG. 4A, vehicle 18 is shown traveling ona straight road equipped with camera 12R by way of example. Objects 54 aand 54 b are shown which may be bicyclists traveling in a bicycle lane.The dotted line emanating from camera 12R shows the extent of thehorizontal effective field of view. Outside the horizontal effectivefield of view, although images of objects Ma and 54 b may be imaged bycamera 12R, algorithm 309 does not detect/classify objects 54 a and 54b. Object 54 b may be partially occluded by the right front pillar ofvehicle 18. Nevertheless, as vehicle 18 is traveling straight, there islittle chance of a collision with objects 54 a and 54 b which arestationary or also traveling straight for instance in the bicycle lane.

The presence of a stationary object such as structural barrier 86 may bedetermined by use of a Structure-from-Motion (SfM) algorithm using imageframes from right camera 12R. SfM may recover three-dimensionalinformation from stationary structural barrier 86. The presence of astationary barrier 86 in the absence of other obstacles such as movingor stationary obstacles 54 a/ 54 b, suppresses warning driver 36 toavoid an unnecessary false positive warning.

In the case where structural barrier 86 may be a sidewalk, a positivewarning may be issued by use of the SfM algorithm. Vehicle 18 turning atan intersection adjacent to the sidewalk, e.g., a right turn from theright most lane, may encroach onto the sidewalk even though the rightfront wheels did not mount the sidewalk. A pedestrian standing on thecorner might be hit. If the bus or truck is turning too sharply then awarning may be issued if a pedestrian is detected on the sidewalk.Detection and classification by the SfM algorithm may indicate thecurvature of the sidewalk at the intersection. If vehicle 18 makes aright turn at the intersection, the area of interest in image frames 15Rresponsive to yaw information (step 307) may be increased to warn driver36 that the present right turn has a potential for the side of vehicle18 to encroach of the sidewalk

Reference is now also made to FIG. 4B showing a road scenario 40 b.Vehicle 18 is shown turning on a right curve, and detection andclassification algorithm 309 is configured to use a wider horizontalangle shown by a dotted line emanating from camera 12R. The widerhorizontal angle corresponds to a wider image portion in horizontalimage coordinates of images frames 15. Again obstacles 54 a and 54 b areshown. Obstacle 54 b may be partially occluded by the right front pillarof vehicle 18. If vehicle 18 is turning right, it may not be clearwhether moving obstacle, e.g bicyclist 54 b is intending to travelstraight at a nearby intersection or if bicyclist 54 b is intending toturn right in parallel with vehicle 18. Even if bicyclist 54 b isintended to turn right, the road condition may be such that bicyclist 54b may unintentionally skid into vehicle 18 during the right turn. Inthis case, detection/recognition of a bicyclist in the effective fieldof view of right camera 12R may cause a warning (step 313 FIG. 3).

In decision block 309, algorithms stored in memory 20C/20R/20L and/orincluded in circuitry of processors 14C/14R/14L may be used to detectand/or recognize obstacles responsive to the yaw information. If anobject is detected and/or recognized then driver 36 is warned (step313).

Reference is now made to FIG. 5A which shows a driver's forward view 50from inside vehicle 18, according to features of the disclosedembodiments Vehicle 18 may be equipped with multiple systems 16 (FIG.2A) each with image frames 15 captured by processor 14 from camera 12.Processor 14 processes image frames 15 and recognizes or detects anobstacle which may collide with vehicle 18.

Alert display 32L is shown to the left of the steering wheel of vehicle18. Alert display 32L connects to an output of processor 14L whichcaptures image frames 15L from camera 12L. An alert output of processor14L may be displayed by display 32L to provide a warning to driver 36 ofvehicle 18. The warning may be of an obstacle recognized by processor14L which is in field of view of left camera 12L where the obstacle isin possible collision with vehicle 18. On the right side of vehicle 18,may be another display 32R connected to an output of processor 14R. Analert output of processor 14R may be displayed by display 32R to providea visible warning to driver 36 of vehicle 18. The warning may resultfrom an obstacle recognized by processor 14R in field of view of rightcamera 12R where the obstacle is in possible collision with vehicle 18.

Alert display 32C is shown positioned in the center at the bottom of thewindshield above the dashboard of vehicle 18. Processor 14C processesimage frames 15C captured from a center camera 12C and recognizes anobstacle with which vehicle 18 may be on a collision course. An outputof processor 14C connects to display 32C to provide a warning to driver36 of vehicle 18. The warning output of processor 14C may be displayedvisually by display 32C. The warning may be of the obstacle recognizedby processor 14C which is visible to center camera 12C.

Three recognized objects are shown marked with rectangular boxes, two ofthe recognized objects are cyclists and the other obstacle is apedestrian. Since the three recognized objects are in front of vehicle18, an appropriate warning may be sounded and/or displayed on one ormore of displays 32R, 32L and/or 32C depending on which of cameras 12L,12R, 12C views the recognized obstacle. Similarly, sounding of anaudible warning may depend on which camera views the recognizedobstacle. If the recognized obstacle is on the right side of vehicle 18,a speaker or buzzer positioned on the right side of the driver may soundthe warning optionally collocated in a housing with right display 32R.

Similarly, if the recognized obstacle is on the left side of vehicle 18,a speaker or buzzer positioned on the left side of the driver may soundthe warning optionally collocated in a housing with left display 32L. Ifthe recognized obstacle is recognized forward and center of vehicle 18,a speaker or buzzer positioned near the driver may sound the warningoptionally collocated in a housing with center display 32C.

Reference is now made to FIGS. 5B, 5C, 5D and 5E which show examples ofdisplay 32C in greater detail, according to features of the disclosedembodiments. FIGS. 5B, 5C, 5D and 5E show four possible warnings whichmay be visibly displayed from display 32C.

With reference to FIG. 5B, display 32C provides a warning to driver 36of vehicle 18 of an imminent rear-end collision with a car, truck ormotorcycle moving at any speed. The warning is shown by a vehicle icon54 in the circular section flashing red for example and/or accompaniedwith an audible warning. If an imminent collision is with a pedestrian,the pedestrian icon 52 may similarly flash red and/or be accompaniedwith an audible warning. The warning of the imminent collision with apedestrian may be only active under a speed limit appropriate for thearea which vehicle 18 is traveling i.e., for example, in a built up areawhere there are pedestrians.

With reference to FIG. 5C, display 32C provides a headway warningaccompanied by a following time displayed as 0.8 seconds. The term“headway” as used herein is defined as a time interval to reach thecurrent position of an obstacle ahead of vehicle 18, and may becalculated by dividing the distance to the obstacle by the travel speedof vehicle 18. If headway in seconds falls below a predefined value, avisible and/or audible warning may displayed to driver 36.

If the obstacle is a lead vehicle or an oncoming vehicle a vehicle icon54 may be used to alert the driver and/or an audible warning togetherwith the display of headway.

With reference to FIG. 5D, display 32C provides a lane departure warning(LDW) shown by dotted line in the circular section of display 32C. Inthe case of FIG. 5D, the dotted line is located on the right side of thecircle to indicate to driver 36 that vehicle 18 has departed lane on theright side without the use of turn signals. Similarly, a departure onthe left hand side of a lane may be warned with a dotted line located onthe left side of the circular section of display 32C. Lane departurewarnings may be displayed with dotted line flashing and/or accompaniedwith an audible warning. The lane departure warning (LDW) may beactivated when vehicle 18 is traveling faster than a threshold speed.LDW may be activated differently if the road is an urban road, highwayor country road. For example in an urban road with bus stops andpedestrians, vehicle 18 often crosses lane markings and lane departurewarning may be activated differently or suppressed.

Reference is now is made to FIG. 5E, display 32C provides a visibleand/or audible warning if driver 36 exceeds the posted speed limit. Anexample posted speed limit in this case is shown on display 32C as 20miles per hour or 20 kilometers per hour.

Reference is now is made to FIG. 5F which shows displays 30L/30R ingreater detail, according to a feature of the disclosed embodiments. Thewarnings provided on displays 30L/30R are as a result of a recognitionof an obstacle by processors 14L/14R. Processors 14L/14R process imageframes 15L/15R captured by cameras 12L/12R and a warning may be issuedfrom the processor 14L/14R and displayed on respective displays 32L/32Raccording to the description that follows.

Displays 30L/30R include pedestrian icon 52. A flashing or solid yellowdisplay of pedestrian icon 52 may warn driver 36 of vehicle 18 turningleft or right of a potential collision with an obstacle. The flashing orsolid yellow display of pedestrian icon 52 indicates to driver 36 that apedestrian or cyclist is near but is not within a collision trajectoryof vehicle 18. However, a flashing or solid red display of pedestrianicon 52 and/or audible sound, on respective displays 30L/30R may warndriver 36 of vehicle 18 that a pedestrian or cyclist is within acollision trajectory of vehicle 18 as vehicle 18 is turning left orright.

Reference is now made to FIG. 6, a flowchart for a method 601, accordingto aspects of the disclosed embodiments. In the discussion that follows,reference is made to camera 12L, processor 14L and display 32L. However,the method steps as follows are similar for other systems 16 of: cameras12R, 12C; processors 14R,14C; and displays 32R, 32C. Method 601 may beperformed in parallel by each of systems 16. In step 603, several imageframes 15 captured from camera 12L by processor 14L are processed sothat in step 605 a candidate image is detected and/or recognized as anobstacle in the field of view of camera 12L. In step 607 a decision ismade if an obstacle detected and/or recognized in step 605 is in thetrajectory of travel of vehicle 18 such that vehicle 18 is likely tocollide with the obstacle. If the trajectory of travel of vehicle 18 islikely to collide with the obstacle then a visible and/or audiblewarning is provided (step 609) to driver 36 of vehicle 18 via display32L.

Alternatively, as described above, in some embodiments, as analternative to providing a warning to driver 36, system 16 may insteadcause the vehicle (e.g., vehicle 18) to take an action. For example,system 16 may analyze the collected data to determine whether or not thevehicle (e.g., vehicle 18) should take a certain action, and thenautomatically take the determined action without human intervention. Forexample, in response to recognizing that an object is in a possiblecollision with the vehicle, system 16 may automatically control thebraking, acceleration, and/or steering of the vehicle (e.g., by sendingcontrol signals to one or more of a throttling system, a braking system,and a steering system). In yet other embodiments, system 16 may providea visible and/or audible warning to the driver and cause the vehicle totake an action.

Many vehicles have blind spots, which blocks the ability of the operatorof the vehicle to notice hazards at certain areas around the vehicle. Indense urban environments, where Vulnerable Road Users (VRUs), including,for example, pedestrians and cyclists, often share the road withvehicles, blind spots and the inability of an operator to detect VRUs intheir blind spots is a serious problem and can lead to grave results.Blind spots may be a particularly severe problem for operators of largevehicles, such as trucks (or Lorries) and public transport vehicles,especially in urban environments.

Accordingly, in still other embodiments, the disclosed systems andmethods select at least a portion of one or more image frames responsiveto identifying a potential risk of a blind spot. The disclosed systemsand methods may then process the selected portions of the one or moreimage frames to detect, for example, an object in the selected portionsof the image frames. Further in some embodiments, as discussed below,portions of image frames may be selected based on areas in whichpotential blind spots and/or associated hazards are known based on, forexample, data collected by the system in previous drives in a particularlocation (e.g., based on a report, as discussed below in more detail).Further still, in some embodiments, selection of at least a portion ofone or more image frames may be based on a yaw rate of the vehicle andcollected data (e.g., a report, etc.).

For example, in some embodiments, the disclosed systems and methods maybe configured to detect VRUs. Optionally, the systems and methods may beconfigured to detect VRUs while ignoring inanimate objects. In someembodiments, the system can include a master camera and one or moreslave cameras. The master camera can be configured to serve an AdvancedDriver Assistance System (which can be implemented on the processor) tothe operator in addition to detecting VRUs. The slave cameras may beconfigured to receive input from the master camera and can monitor oneor more blind spots around the vehicle.

The processor may process the images from the one or more cameras anddetects a potential hazard in a blind spot. Optionally, the processormay be configured to issue an alert when a hazard is detected in a blindsport area. Optionally, the processor may be configured to initiate anoperation in the vehicle or in one of the vehicle's systems in responseto detecting a hazard in a blind spot of the vehicle.

For example, the system (e.g., system 16) may include a camera controlunit that is configured to adjust an effective coverage of at least onecamera that is mounted on or in the vehicle and which is used to monitora certain area around the vehicle. Still further by way of example, thecamera control unit may be configured to adjust an effective field ofview of the least one camera. In yet another example, the camera controlunit can be configured to rotate the at least one camera so as to adjustits field of view in a desired direction, as necessary. Optionally, thecamera control unit can be implemented in a master camera, and themaster camera may be configured to control and adjust at least one slavecamera. According to a further example, the master camera may beconfigured to control the detection angle of the slave cameras so thatthey are configured to detect VRUs in specific areas (e.g., effectiveFOVs) around the vehicle. In some embodiments, the system may includeone, two, three, four or more cameras. For convenience, and by way ofnon-limiting example, in the following description, the system's camerasare sometime referred to as “a camera array.” The processor may beconfigured to control a first camera in the camera array according to arespective adjustment which the processor applied or is about to applyto a second camera in the camera array. In another example, two or moreof the cameras in the camera array may be adjusted cooperatively and inconcert. Optionally, the adjustment of the cameras may be performed inconnection with a dynamically changing location or with a projectedlocation of one or more blind spots in relation to the vehicle. In someembodiments, the disclosed systems and methods may display a map ofhazards detected by smart cameras and/or of alerts issued by smartcameras onboard vehicles. The term “smart camera” refers to a camera,which captures images, or a video sequence, or an environment, and oneor more processors (and possibly other computer hardware) which processthe images to provide information about objects appearing in the images.Various smart cameras are known and used in driver assistance systemsand in autonomous vehicle systems. According to examples of thepresently disclosed subject matter, the method can include: receiving aplurality of reports of hazards detected by smart cameras and/or ofalerts issued by smart cameras onboard vehicles and storing them, eachone of the plurality of reports includes at least a locationinformation, where the location information is a location in a globalcoordinate of a respective hazard detected by a smart camera and/or of arespective alert issued by one or more smart camera onboard a vehicle;receiving a display parameter and providing indication on the map ofhazards detected by smart cameras and/or of alerts issued by smartcameras on board vehicles according to the display parameter. In afurther example, the method can include, based on the display parameter,aggregating two or more reports of hazards detected by smart camerasand/or of alerts issued by smart cameras onboard vehicles, according tothe display parameter.

Still further by way of example, the display parameter may be a regionof interest which indicates which portion of the map is to be displayedon a display screen to the user. Further by way of example, theindications on the map of hazards detected by smart cameras and/or ofalerts issued by smart cameras onboard vehicles are indications ofhazards detected and/or of alarms issued within the region of interest.Further by way of example, the region of interest can be adjusteddynamically, and the display of the hazards detected and/or of alarmsissued may also change dynamically and respectively. For example, thedisplay parameter may be a scale of the map. In yet further examples,hazards detected by smart cameras and/or of alerts issued by smartcameras onboard vehicles may be aggregated according to a scale of themap. Thus for example, when the scale of the map is relatively large,detected hazards and/or issued alerts from a relatively large area maybe combined and a respective indication of the aggregate detectedhazards and/or issued alerts may be displayed on the map in a locationwhich is, for example, a center location of all the detected hazardsand/or issued alerts, whereas when the scale of the map is relativelysmall, detected hazards and/or issued alerts from a relatively smallarea are combined and a respective indication of the aggregate detectedhazards and/or issued alerts is displayed on the map. It would beappreciated that this feature can enable a dynamic display of at least alocation aspect of detected hazards and/or issued alerts. Thus forexample, a user can start with a map of an entire metropolitan area,viewing a general layout of detected hazards and/or issued alerts, andby zooming in, the user can gradually (although it can also be in anon-gradual manner) hone in on specific areas where a large number ofalerts were triggered in a single location.

For example, when a plurality of detected hazards and/or issued alertsare reported for a single location or for a certain area, an indicationmay be displayed at the respective location of the map, and theindication can be associated with the number or density of the reporteddetected hazards and/or issued alerts. In another example, theindication can also be associated with a severity of the detectedhazards and/or issued alerts. In still further examples, thresholds canbe set, for example, by a user, or predefined threshold can be provided,to set forth conditions when different types or different information isdisplayed on the map in association with respective detected hazardsand/or issued alerts In still further examples, the indications on themap may relate to reported detected hazards and/or issued alerts canalso be associated with a respective type of each detected hazard and/orissued alert. In a further example, detected hazards and/or issuedalerts may be aggregated together. In still further examples, a counterwhich displays a number of events or reports (for example a total numberof detected hazards and/or issued alerts) may be updated according to anumber of detected hazards and/or issued alerts of the same type thatare associated with a certain location or area in the map. A size of anarea associated with a counter, and optionally its location, may beassociated with a scale of the map or with an area shown in the map. Ina digital map, the scale and area of reference may be dynamicallyadjusted, and as the scale changes, so could the display of the detectedhazards and/or issued alerts.

In some embodiments, the disclosed systems and methods may generate asignature of hazards detected by smart cameras and/or of alerts issuedby smart cameras onboard vehicles. As mentioned above, one or morecameras and at least one processor may be mounted on a vehicle. Theprocessor may process the images from the camera and detect hazardsaround the vehicle, which may trigger alerts and/or operations in thevehicle or in one of the vehicle's systems. For example, the processormay be configured to issue a report with respect to a detected hazard ina vicinity of the vehicle. The processor may cause the report to becommunicated to a backend server, or the processor may locally store thereports (e.g., in a memory onboard the vehicle) and the backend servermay be configured to ‘pull’ such reports. Optionally, a report mayinclude a location of a hazard and a time when the hazard was detectedor when an incident that is associated with the hazard occurred (e.g.,when the incident was captured by the onboard camera). Optionally, thereport can include information about the type of hazard that wasdetected. In addition or as an alternative to the type of hazardindication, the report can include one or more images that areassociated with the detected hazard, including images of an incident ora potential incident which triggered the alert or the operation imagesof the road and/or of the surrounding environment and infrastructure,and possibly information about the camera that was used to capture theimages.

In some embodiments, the server may be configured to process incomingreports from smart cameras onboard vehicles. Optionally, the server canbe configured to identify related reports, for example, reports that areassociated with a common or with a related safety hazard, and the servercan generate a signature in respect of a certain safety hazard. Eachsignature can be associated with one or more reports of a safety hazard.Further, for example, the disclosed systems and methods may identifysimilar safety hazards that are associated with hazards detected by asmart camera mounted on a vehicle. In one example, a matching algorithmmay be used to determine if two different data sets, when compared,yield a result that is within a predefined threshold or range. Thus forexample, an indication of similarity of a safety hazard between a firstset of reports and a second set of reports can be concluded when in thefirst set of reports, the area or the point where an object thattriggered a hazard detection and/or alert was located relative to thevehicle at a certain location or at a certain set of locations thatis/are identical or close enough (according to some proximity threshold)to the respective location or area in a second set of reports.

In another example, image processing, geometrical representation,schematics and the like may be used to determine similarity, such that,for example, when two environments have an identical or close enough(according to some similarity threshold) layout. The layout may be ageneral layout of the street, or even just a position of single object.For example, it can be determined in one location where a relativelylarge number of hazards were detected or alerts where issued, that acertain sign on the sidewalk blocked the line of sight of a bus whoseroute passed by the sign, as well as the line of sight of pedestrianswhich crossed the street near the sign. This ill-positioned signedcaused a large number of hazard detections or alerts. The reportsindicated the points relative to the vehicle where the hazard wasdetected or the alerts were issued, and also a position of the inanimateobject relative to the bus, or the sign in this particular example.Sometime later, a similar situation at a different location (forexample, a different street, a different city, or even a differentcountry) caused a similar hazard detections or alerts pattern, and thereports may have indicated an inanimate object that is located at asimilar location relative to the vehicle.

In some embodiments, the system may include a memory that is capable ofstoring a plurality of safety hazard entries, where each safety hazardentry includes a signature. Each signature may include structured datawhich characterizes the underlying safety hazard. The system may furtherinclude a processor that is configured to receive a first safety hazardentry, extract from the first safety hazard entry a first signature, andgenerate a search query based on the first signature. According to oneexample, the search query may include at least one acceptable range thatis based on at least a first data item in the first signature, and theprocessor may be configured to retrieve from the memory at least asecond signature that includes at least a second data item that iswithin the predefined range. In another example, the processor may beconfigured to retrieve any signature that is within the acceptablerange.

In some embodiments, the disclosed systems and methods may select and/orsuggest a reaction to a first safety hazard based on a similaritybetween a signature with which the first safety hazard is associated anda signature with which a second safety hazard is associated, where thesuggested reaction to the first safety hazard is a reaction that wasapplied to the second safety hazard. The reaction may be a simulated orvirtual reaction, or an actual physical measure that was taken in anattempt to mitigate or eliminate the hazard.

The system can include, by way of example, memory and a processor. Thememory may hold a safety hazard entry. Optionally, the safety hazardentry can include a signature of a detected hazard and/or issued alertreported by smart cameras on board a vehicle. For example, a signaturemay be associated with a plurality of hazard detections and/or with aplurality of issued alerts, which plurality of detected hazards and/orissued alerts were determined to be associated with one another. Theplurality of hazard detections and/or with a plurality of issued alertsthat are associated with the signature may have originated from aplurality of vehicles (each having a smart camera onboard). For example,the detected hazards and/or issued alerts which are associated with acertain signature can be hazards or alerts which have been determined tobe associated with a specific cause or with a plurality of relatedcauses. Optionally, some or all of the data may be structured, such thatit can be searched, evaluated, co-processed with data from other safetyhazard entries (e.g., combined, compared, etc.).

Further by way of example, the safety hazard entry may be associatedwith a specific reaction or with related reactions. Optionally, a safetyhazard entry may be associated with a single reaction, or in some casesand according to some examples of the presently disclosed subjectmatter, a safety hazard entry can be associated with more than onereaction (e.g. with two, three or more reactions). Optionally, thereaction data may be structured, such that it can be searched,evaluated, co-processed with data from other safety hazard entries(e.g., combined, compared, etc.). The reaction data may be related to achange that was made in the road and/or in its surroundings. Optionally,in association with the reaction data there can be stored an indicationof an efficacy of the underlying reaction, and possibly of other effectsand tradeoffs, such as: if any parking areas were cancelled to providethe reaction, if congestion increased as a result, if a new hazard wascreated or an existing one was exacerbated, and possibly also to whatextent, etc.

Optionally, the reports may be processed to determine the root cause orcauses and/or the contributing factors or circumstances which contributeto the safety hazard. In another example, the root cause or causesand/or the contributing factors or circumstances which contribute to thesafety may be provided manually by a user. For example, the processorcan be configured to provide the structured signature based on theprocessing of the reports from smart cameras onboard vehicles. Accordingto examples of the presently disclosed subject matter, a reaction reportmay also be stored in the memory. The reaction report may be associatedwith one of more safety hazards. In one example, the reaction report maybe associated with a certain safety hazard signature, and thus anysafety hazard entry whose signature matches or is similar enough to thesignature with which the reaction report is associated, may becomeassociated with the reaction report.

By way of example, by providing a signature of a yet to be resolvedsafety hazard, it may be possible to provision or select a solution tothe yet to be resolved safety hazard based on a similarity between itssignature and the signature or one or more previously resolved safetyhazards. The use of the term previously resolved should not be construedas binding the examples in the present disclosure the solutions whichare necessarily successful in resolving the underlying problem, and thatin many cases the solutions can be partially successful, and in othercases there could be a tradeoff for a given solution, and in variouscircumstances the tradeoff may be more or less appealing, and evenunacceptable, and indeed in certain examples of the presently disclosedsubject matter, more than one solution can be returned for a givensignature, and the processor can be configured to select from theplurality of solutions returned for a given signature the ones or onewhich provides the based tradeoff or which does not contradict one ormore constraints with which the respective signature of the subjectunresolved safety hazard is associated.

Various methods may be performed by systems mountable in a vehicle toprovide object detection in the vicinity of the vehicle. The systemsinclude multiple cameras operatively attached to one or more processors.A right camera is mounted externally on the right side at or toward therear of the vehicle and/or a left camera is mounted externally on theleft side at or toward the rear of the vehicle. The fields of view ofthe cameras are substantially in the forward direction of travel of thevehicle. A right warning device is mountable on the vehicle to the rightof the driver and a left warning device is mountable on the vehicle tothe left of the driver. The right warning device and the left warningdevice are connectable to the one or more processors.

Multiple image frames may be captured from the cameras. The image framesmay be processed to detect a moving object in a portion of the imageframes. It may be determined that a potential collision of the vehiclewith the moving object has a likelihood greater than a threshold and awarning may be issued either by sounding an audible warning ordisplaying a visual warning to a driver of the vehicle responsive to thelikelihood of the potential collision with the moving object. When thedetected object is viewed by the right camera, the warning may issuefrom a right warning device mounted on the vehicle to the right of thedriver and when the detected object is viewed by the left camera, thewarning may issue from a left warning device mounted on the left side ofthe vehicle to the left of the driver.

The system may further include a center camera mounted in front of thedriver and a central processor connected to the central camera and aforward warning device connected to the central processor which warns ofan impending collision with a moving obstacle essentially in the forwarddirection of the vehicle.

The system may further include using another driver assistance systeminstalled such as lane detection, structural barrier recognition and/ortraffic sign recognition to recognize a stationary object. Any warningto the driver may be suppressed if the other driver assistance systemdetermined that all objects detected in the immediate vicinity of thevehicle are stationary objects.

Various methods are provided herein for using a system mountable on avehicle to provide object detection in the vicinity of the vehicle. Thesystem includes a camera operatively attached to a processor. The camerais mounted at the rear of the vehicle. The field of view of the camerais substantially in the forward direction of travel of the vehicle alonga side of the vehicle. Multiple image frames are captured from thecamera. An intersection is detected while the vehicle is turning. Apotential encroachment of the vehicle onto the sidewalk is determined atthe intersection by the vehicle if the vehicle turns too sharply at theintersection. The contour of the curb of the sidewalk may be detected atthe intersection by detecting a contour of the curb of the side-walk atthe intersection using a structure from motion algorithm which processesimage motion over time of the image frames. An obstacle, e.g., object orpedestrian may be detected on the sidewalk. A warning may be issued ifthe bus is determined to encroach the sidewalk and/or warning may beinhibited if the bus is determined not to encroach the sidewalk.

The terms “sidewalk” and “pavement” are used herein interchangeably. Theterms “object and obstacle” are used herein interchangeably. The term“rear” as used herein in the context of the rear of a vehicle refers tothe rear half of the vehicle near to the back or the rear passenger doorof a bus. The terms “right” and “left” are from the point of view of adriver facing forward of a vehicle driving essentially in the forwarddirection of the vehicle. The term “corresponding” as used herein refersto matching image points in different image frames which are found to beof the same object point. The indefinite articles “a” and “an” is usedherein, such as “a processor”, “a camera”, “an image frame” have themeaning of one or more that is “one or more processors”, “one or morecameras” or “one or more image frames.”

Although exemplary embodiments have been shown and described, it is tobe understood these examples are not limiting. Instead, it is to beappreciated that changes may be made to these embodiments, the scope ofwhich is defined by the claims and the equivalents thereof.

1. A method performed by a system mountable in a vehicle to provideobject detection in the vicinity of the vehicle, the system including acamera operatively attached to a processor, wherein the camera ismounted externally at the rear of the vehicle, wherein the field of viewof the camera is substantially in the forward direction of travel of thevehicle along a side of the vehicle, the method comprising: capturing aplurality of image frames with the camera; selecting respective portionsof the image frames responsive to yaw of the vehicle; and processing theimage frames thereby detecting an object imaged in the selected portionsof the image frames.
 2. The method of claim 1, wherein the yaw is inputfrom at least one device selected from the group of devices consistingof: the camera by processing the image frames, a gyroscopic device, turnsignals, steering angle and a sensor attached to the steering column ofvehicle.
 3. The method of claim 1, the method further comprising: uponmeasuring an increase in absolute value of the yaw, processing a largerarea of the image frames thereby increasing an effective horizontalfield of view to detect an obstruction over a wider horizontal anglemeasured from the left side of the vehicle.
 4. The method of claim 1,wherein the camera is installed externally on the right side of thevehicle, the method further comprising: upon the vehicle turning right,processing a larger area of the image frames thereby increasing aneffective horizontal field of view to detect an obstruction over a widerhorizontal angle measured from the right side of the vehicle.
 5. Themethod of claim 1, wherein the camera is installed externally on theleft side of the vehicle, the method further comprising: upon thevehicle turning left, processing a larger area of the image framesthereby increasing an effective horizontal field of view to detect anobstruction over a wider horizontal angle measured from the left side ofthe vehicle.
 6. The method of claim 1, further comprising: determiningsaid object is a moving obstacle; verifying that said object will be ina projected path of the vehicle thereby determining a potentialcollision of said vehicle with said moving obstacle; and warning adriver of said vehicle of said potential collision with said movingobstacle.
 7. The method of claim 6, wherein said warning is selectedfrom the group consisting of: sounding an audible warning and adisplaying a visible warning.
 8. The method of claim 7, furthercomprising: issuing said warning from the side of the vehicle on whichis mounted the camera viewing the moving obstacle.
 9. The method ofclaim 1, wherein said object is selected from a group of objectsconsisting: a bicyclist, a motorcyclist, a vehicle, a pedestrian, achild riding on a toy vehicle, a person with pram and a wheelchair user.10. The method of claim 1, further comprising: using another driverassistance system selected from the group consisting of: lane detection,structural barrier recognition, traffic sign recognition to recognize astationary object; and enabling suppression of a warning to the driverif all objects detected in the immediate vicinity of the vehicle arestationary objects.
 11. A system mountable in a vehicle to provideobject detection in the vicinity of the vehicle, the system including acamera operatively attached to a processor, wherein the camera ismounted externally at the rear of the vehicle, wherein the field of viewof the camera is substantially in the forward direction of travel of thevehicle along a side of the vehicle; the system operable to: capture aplurality of image frames with the camera; select respective portions ofthe image frames responsive to yaw of the vehicle; and process the imageframes thereby detecting an object imaged in the selected portions ofthe image frames; and detect an object imaged in the selected portionsof the image frames.
 12. The system of claim 11, wherein the yaw ismeasurable responsive to at least one device selected from the group ofdevices consisting of: the camera by processing the image frames, agyroscopic device, turn signals, steering angle and a sensor attached tothe steering column of vehicle.
 13. The system of claim 11, furtheroperable to: upon measuring an increase in absolute value of the yaw,process a larger area of the image frames to increase an effectivehorizontal field of view and to detect thereby an obstruction over awider horizontal angle measured from the side of the vehicle.
 14. Thesystem of claim 11, wherein the camera is installed on the right side ofthe vehicle, the method further operable to: when the vehicle turnsright, process a larger area of the image frames to increase aneffective horizontal field of view to detect thereby an obstruction overa wider horizontal angle measured from the right side of the vehicle.15. The system of claim 11, wherein the camera is installed on the leftside of the vehicle, the method further operable to: when the vehicleturns left, process a larger area of the image frames to increase aneffective horizontal field of view and to detect thereby an obstructionover a wider horizontal angle measured from the left side of thevehicle.
 16. The system of claim 11, further operable to: determine saidobject is a moving obstacle; verify that said object will be in aprojected path of the vehicle thereby determining a potential collisionof said vehicle with said moving obstacle; and warn a driver of saidvehicle of said potential collision with said moving obstacle.
 17. Thesystem of claim 16, wherein said warning is selected from the groupconsisting of: sounding an audible warning and a displaying a visiblewarning.
 18. The system of claim 17, further operable to: issue saidwarning from the side of the vehicle on which is mounted the cameraviewing the moving obstacle.
 19. The system of claim 11, wherein saidobject is selected from a group of objects consisting: a bicyclist, amotorcyclist, a vehicle, a pedestrian, a child riding on a toy vehicle,a person with pram and a wheelchair user.
 20. The system of claim 11,further operable to: using another driver assistance system selectedfrom the group consisting of: lane detection, structural barrierrecognition, traffic sign recognition to recognize a stationary object;and suppress a warning to the driver if all objects detected in theimmediate vicinity of the vehicle are stationary objects.